# Logstash Problem With Open Distro Docker

**URL:** <https://forum.opensearch.org/t/logstash-problem-with-open-distro-docker/2643>\
**Category:** Open Source Elasticsearch and Kibana\
**Created:** [April 23, 2020, 2:46am UTC](https://forum.opensearch.org/t/logstash-problem-with-open-distro-docker/2643 "2020-04-23T02:46:42Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![rkredux](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@rkredux](https://forum.opensearch.org/u/rkredux)\
**Post date:** [April 23, 2020, 2:46am UTC](https://forum.opensearch.org/t/logstash-problem-with-open-distro-docker/2643/1 "2020-04-23T02:46:43Z")

</div>

My docker-compose file (taken from Opendistro documentation - [Docker - Open Distro Documentation](https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/)

```
version: '3'
services:
  odfe-node1:
image: amazon/opendistro-for-elasticsearch:1.6.0
container_name: odfe-node1
environment:
  - cluster.name=odfe-cluster
  - node.name=odfe-node1
  - discovery.seed_hosts=odfe-node1,odfe-node2
  - cluster.initial_master_nodes=odfe-node1,odfe-node2
  - bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
  - "ES_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
ulimits:
  memlock:
    soft: -1
    hard: -1
  nofile:
    soft: 65536 # maximum number of open files for the Elasticsearch user, set to at least 65536 on modern systems
    hard: 65536
volumes:
  - odfe-data1:/usr/share/elasticsearch/data
ports:
  - 9200:9200
  - 9600:9600 # required for Performance Analyzer
networks:
  - odfe-net
  odfe-node2:
image: amazon/opendistro-for-elasticsearch:1.6.0
container_name: odfe-node2
environment:
  - cluster.name=odfe-cluster
  - node.name=odfe-node2
  - discovery.seed_hosts=odfe-node1,odfe-node2
  - cluster.initial_master_nodes=odfe-node1,odfe-node2
  - bootstrap.memory_lock=true
  - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
ulimits:
  memlock:
    soft: -1
    hard: -1
  nofile:
    soft: 65536
    hard: 65536
volumes:
  - odfe-data2:/usr/share/elasticsearch/data
networks:
  - odfe-net
  kibana:
image: amazon/opendistro-for-elasticsearch-kibana:1.6.0
container_name: odfe-kibana
ports:
  - 5601:5601
expose:
  - "5601"
environment:
  ELASTICSEARCH_URL: https://odfe-node1:9200
  ELASTICSEARCH_HOSTS: https://odfe-node1:9200
networks:
  - odfe-net
  logstash:
image: docker.elastic.co/logstash/logstash-oss:7.3.2
container_name: logstash
volumes:
  - ./logstash.docker.conf:/usr/share/logstash/pipeline/logstash.conf
  - .:/usr/share/logstash/logs
ports:
  - "5011:5011"
  - "5012:5012"
  - "5013:5013"
networks:
  - odfe-net
depends_on:
  - odfe-node1

volumes:
  odfe-data1:
  odfe-data2:

networks:
  odfe-net:

```

My logstash pipeline conf file

```
input {
  http {
    port => 5011
  }
  udp {
    port => 5012
    codec => "json"
  }
  tcp {
    port => 5013
    codec => "json_lines"
  }
}
filter {
  json{
    source => "message"
  }
  
  elasticsearch {
    hosts => ["odfe-node1:9200", "odfe-node2:9200"]
    query => "name:%{name}"
    fields => { "spend" => "current_spend" }
    index => "logstash-etl"
    user => "logstash"
    password => "logstash"
    enable_sort => false 
    ssl => false 
  }

  ruby {
    code => "event.set('spend', (event.get('current_spend') + event.get('spend')))"
    remove_field => ["current_spend"]
  }

  fingerprint {
    source => "name"
    target => "[@metadata][fingerprint]"
    method => "MURMUR3"
  }
}

output {
  elasticsearch {
    hosts => ["https://odfe-node1:9200","https://odfe-node2:9200"]
    ssl => false
    ssl_certificate_verification => false
    user => logstash
    password => logstash
    ilm_enabled => false
    index => "logstash-etl"
  }
  stdout {
  }
}

```

**My error log in Logstash (which I don’t see if I remove the Elasticsearch query filter)**

`logstash | [2020-04-23T02:21:31,017][ERROR][logstash.javapipeline] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<Manticore::SocketException: Connection refused (Connection refused)>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:37:in `block in initialize’“, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:79:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:274:in `call\_once’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/manticore-0.6.4-java/lib/manticore/response.rb:158:in `code'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `block in perform\_request’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/base.rb:262:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform\_request’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-transport-5.0.5/lib/elasticsearch/transport/client.rb:131:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/elasticsearch-api-5.0.5/lib/elasticsearch/api/actions/ping.rb:20:in `ping’”, “/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.6.0/lib/logstash/filters/elasticsearch.rb:192:in `test_connection!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-elasticsearch-3.6.0/lib/logstash/filters/elasticsearch.rb:74:in `register’”, “org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:56:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:192:in `block in register\_plugins’”, “org/jruby/RubyArray.java:1792:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191:in `register\_plugins’”, “/usr/share/logstash/logstash-core/lib/logstash/java\_pipeline.rb:463:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:204:in `start\_workers’”, “/usr/share/logstash/logstash-core/lib/logstash/java\_pipeline.rb:146:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:105:in `block in start’”], :thread=\>”#\<Thread:0x5d027f32 run\>"}`

**Error from one of the Elasticsearch node**

```
odfe-node1 | [2020-04-23T02:20:16,456][ERROR][c.a.o.s.s.h.n.OpenDistroSecuritySSLNettyHttpServerTransport] [odfe-node1] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 48454144202f20485454502f312e310d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a417574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a436f6e74656e742d547970653a206170706c69636174696f6e2f6a736f6e0d0a486f73743a206f6466652d6e6f6465313a393230300d0a557365722d4167656e743a204d616e7469636f726520302e362e340d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a0d0a
odfe-node1 | io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 48454144202f20485454502f312e310d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a417574686f72697a6174696f6e3a204261736963205957527461573436595752746157343d0d0a436f6e74656e742d547970653a206170706c69636174696f6e2f6a736f6e0d0a486f73743a206f6466652d6e6f6465313a393230300d0a557365722d4167656e743a204d616e7469636f726520302e362e340d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a0d0a

```

Even though ssl is set to false in the Elasticsearch filter plugin, it still throws an error. Any one know why this could be happening?

---

<div class="post-metadata">

**Author:** ![rkredux](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@rkredux](https://forum.opensearch.org/u/rkredux)\
**Post date:** [April 23, 2020, 2:47am UTC](https://forum.opensearch.org/t/logstash-problem-with-open-distro-docker/2643/2 "2020-04-23T02:47:59Z")

</div>

This problem is still there even if I use the latest Logstash image logstsash:7.6.2

---

<div class="post-metadata">

**Author:** ![ogulman](https://avatars.discourse-cdn.com/v4/letter/o/9fc29f/32.png) [@ogulman](https://forum.opensearch.org/u/ogulman)\
**Post date:** [April 23, 2020, 9:53am UTC](https://forum.opensearch.org/t/logstash-problem-with-open-distro-docker/2643/3 "2020-04-23T09:53:27Z")

</div>

You should also specify **http** in uri:

 ![image](https://us1.discourse-cdn.com/flex019/uploads/mauve_hedgehog/original/1X/ef524c262f76ff042824895c132cf75b40f2c0d5.png)

---

<div class="post-metadata">

**Author:** ![rkredux](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@rkredux](https://forum.opensearch.org/u/rkredux)\
**Post date:** [April 23, 2020, 10:49am UTC](https://forum.opensearch.org/t/logstash-problem-with-open-distro-docker/2643/4 "2020-04-23T10:49:29Z")

</div>

@ogulman Thanks. I changed that it to `http` but still get the error below. As I said, the only way this problem goes away is if I remove the `Elasticsearch` filter plugin used in the Logstash pipeline. I don’t get why this happens even though I set `ssl => false` in the plugin setting.

```
logstash | Apr 23, 2020 10:40:58 AM org.apache.http.impl.execchain.RetryExec execute
logstash | INFO: I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://odfe-node1:9200: The target server failed to respond
logstash | Apr 23, 2020 10:40:58 AM org.apache.http.impl.execchain.RetryExec execute
logstash | INFO: Retrying request to {}->http://odfe-node1:9200
odfe-node1 | [2020-04-23T10:40:58,372][ERROR][c.a.o.s.s.h.n.OpenDistroSecuritySSLNettyHttpServerTransport] [odfe-node1] Exception during establishing a SSL connection: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 48454144202f20485454502f312e310d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a417574686f72697a6174696f6e3a204261736963206247396e63335268633267366247396e633352686332673d0d0a436f6e74656e742d547970653a206170706c69636174696f6e2f6a736f6e0d0a486f73743a206f6466652d6e6f6465313a393230300d0a557365722d4167656e743a204d616e7469636f726520302e362e340d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a0d0a
odfe-node1 | io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 48454144202f20485454502f312e310d0a436f6e6e656374696f6e3a204b6565702d416c6976650d0a417574686f72697a6174696f6e3a204261736963206247396e63335268633267366247396e633352686332673d0d0a436f6e74656e742d547970653a206170706c69636174696f6e2f6a736f6e0d0a486f73743a206f6466652d6e6f6465313a393230300d0a557365722d4167656e743a204d616e7469636f726520302e362e340d0a4163636570742d456e636f64696e673a20677a69702c6465666c6174650d0a0d0a

```

---

<div class="post-metadata">

**Author:** ![goodbyepc](https://avatars.discourse-cdn.com/v4/letter/g/f1d935/32.png) [@goodbyepc](https://forum.opensearch.org/u/goodbyepc)\
**Post date:** [April 24, 2020, 4:08am UTC](https://forum.opensearch.org/t/logstash-problem-with-open-distro-docker/2643/5 "2020-04-24T04:08:48Z")

</div>

try~

```auto
elasticsearch {
    hosts => "elasticsearch"
    index => "logstash-%{+YYYY.MM.dd}"
    ssl=>true
    ssl_certificate_verification => false
    ilm_enabled => false
    user=>"user"
    password=>"passwd"
}
```