# Logstash exited with code 1

**URL:** <https://forum.opensearch.org/t/logstash-exited-with-code-1/7123>\
**Category:** Open Source Elasticsearch and Kibana\
**Tags:** security-issue\
**Created:** [September 28, 2021, 3:08am UTC](https://forum.opensearch.org/t/logstash-exited-with-code-1/7123 "2021-09-28T03:08:17Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![travlest](https://avatars.discourse-cdn.com/v4/letter/t/e274bd/32.png) [@travlest](https://forum.opensearch.org/u/travlest)\
**Post date:** [September 28, 2021, 3:08am UTC](https://forum.opensearch.org/t/logstash-exited-with-code-1/7123/1 "2021-09-28T03:08:17Z")

</div>

Hi,

I am new to OpenSearch. I already have elastic and kibana up and running… now I want to pull log from api (using http poller) and push the log from logstash to elastic… but having this kind of problem.

```auto
logstash | [2021-09-28T02:53:05,233][INFO][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
logstash | [2021-09-28T02:53:05,245][INFO][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
logstash | [2021-09-28T02:53:05,590][INFO][logstash.agent] No persistent UUID file found. Generating new UUID {:uuid=>"bf4885fc-a55a-4c28-895a-a885cc4db297", :path=>"/usr/share/logstash/data/uuid"}
logstash | [2021-09-28T02:53:06,689][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
logstash | [2021-09-28T02:53:07,097][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"if\", [A-Za-z0-9_-], '\"', \"'\", \"}\" at line 30, column 1 (byte 638) after output {\n opensearch {\n hosts => [\"https://odfe-node1:9200\"]\n index => \"logstash-index-test\"\n user => \"admin\"\n password => \"admin\"\n ssl => true\n cacert => \"./logstash_pipeline/root-ca.pem\"\n ilm_enabled => false\n }\n\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:187:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:384:in `block in converge_state'"]}
logstash | [2021-09-28T02:53:07,194][INFO][logstash.runner] Logstash shut down.
logstash | [2021-09-28T02:53:07,204][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
logstash | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash | at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.19.0.jar:?]
logstash | at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.19.0.jar:?]
logstash | at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
logstash exited with code 1

```

**my docker compose file**

```auto
version: '3.7'
services:
  odfe-node1:
    image: amazon/opendistro-for-elasticsearch:1.13.2
    container_name: odfe-node1
    environment:
      - cluster.name=odfe-cluster
      - node.name=odfe-node1
      - discovery.seed_hosts=odfe-node1
      - cluster.initial_master_nodes=odfe-node1
      - bootstrap.memory_lock=true #
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m" #
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65536 #
        hard: 65536
    volumes:
      - odfe-data1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9600:9600 # required for Performance Analyzer
    networks:
      - odfe-net
  kibana:
    image: amazon/opendistro-for-elasticsearch-kibana:1.13.2
    container_name: odfe-kibana
    ports:
      - 5601:5601
    expose:
      - "5601"
    environment:
      ELASTICSEARCH_HOSTS: https://odfe-node1:9200
    networks:
      - odfe-net
  logstash:
    image: docker.elastic.co/logstash/logstash-oss:8.0.0-alpha2
    container_name: logstash
    volumes:
    - type: bind
      source: ./logstash_pipeline/
      target: /usr/share/logstash/pipeline
      read_only: true
    ports:
      - "5044:5044"
      - "80:80"
    networks:
      - odfe-net
    depends_on:
      - odfe-node1

volumes:
  odfe-data1:

networks:
  odfe-net:

```

\*\* Logstash confiugration\*\*

```auto
input{
  http_poller {
    urls => {
      authentication => {
        method => get
        url => "webapi/api/"
        headers => {
          "Content-Type" => "application/json"
        }
      }
    }
    request_timeout => 60
    schedule => { every => "5s"}
    codec => "json"
  }
}
output {
  opensearch {
    hosts => ["https://odfe-node1:9200"]
    index => "logstash-index-test"
    user => "admin"
    password => "admin"
    ssl => true
    cacert => "./logstash_pipeline/root-ca.pem"
    ilm_enabled => false
  }
}

```

can anybody help me with it?  
thank you

---

<div class="post-metadata">

**Author:** ![searchymcsearchface](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/searchymcsearchface/32/868_2.png) [@searchymcsearchface](https://forum.opensearch.org/u/searchymcsearchface)\
**Post date:** [September 28, 2021, 2:47pm UTC](https://forum.opensearch.org/t/logstash-exited-with-code-1/7123/2 "2021-09-28T14:47:46Z")

</div>

You can’t use that version of Logstash - you need the OpenSearch Output Plugin.

> **[Logstash](https://opensearch.org/docs/latest/clients/logstash/index/)**
>
> Logstash

---

<div class="post-metadata">

**Author:** ![travlest](https://avatars.discourse-cdn.com/v4/letter/t/e274bd/32.png) [@travlest](https://forum.opensearch.org/u/travlest)\
**Post date:** [September 29, 2021, 2:56am UTC](https://forum.opensearch.org/t/logstash-exited-with-code-1/7123/3 "2021-09-29T02:56:51Z")

</div>

Oh okay.

May I know, how the structure for logstash if I would like to send events to Logstash through HTTP? Is it the same or need changes?

Thank you.

---

<div class="post-metadata">

**Author:** ![searchymcsearchface](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/searchymcsearchface/32/868_2.png) [@searchymcsearchface](https://forum.opensearch.org/u/searchymcsearchface)\
**Post date:** [September 29, 2021, 1:07pm UTC](https://forum.opensearch.org/t/logstash-exited-with-code-1/7123/4 "2021-09-29T13:07:25Z")

</div>

Are you talking about logstash-\>OpenSearch/Open Distro?

---

<div class="post-metadata">

**Author:** ![travlest](https://avatars.discourse-cdn.com/v4/letter/t/e274bd/32.png) [@travlest](https://forum.opensearch.org/u/travlest)\
**Post date:** [September 30, 2021, 12:53am UTC](https://forum.opensearch.org/t/logstash-exited-with-code-1/7123/5 "2021-09-30T00:53:44Z")

</div>

Yes, for logstash configuration.

```auto
input{
  http_poller {
    urls => {
      authentication => {
        method => get
        url => "webapi/api/"
        headers => {
          "Content-Type" => "application/json"
        }
      }
    }
    request_timeout => 60
    schedule => { every => "5s"}
    codec => "json"
  }
}

```

These configurations are wrong, so which structure is the right one?.

---

<div class="post-metadata">

**Author:** ![searchymcsearchface](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/searchymcsearchface/32/868_2.png) [@searchymcsearchface](https://forum.opensearch.org/u/searchymcsearchface)\
**Post date:** [October 1, 2021, 3:27pm UTC](https://forum.opensearch.org/t/logstash-exited-with-code-1/7123/6 "2021-10-01T15:27:48Z")

</div>

Looking more deeply at your error message vs what you posted I noticed that your posted config is 27 lines long and the error message is complaining about an unexpected character on line 30. Do you have some sort of extra character in your config file?

---

<div class="post-metadata">

**Author:** ![travlest](https://avatars.discourse-cdn.com/v4/letter/t/e274bd/32.png) [@travlest](https://forum.opensearch.org/u/travlest)\
**Post date:** [October 4, 2021, 3:48am UTC](https://forum.opensearch.org/t/logstash-exited-with-code-1/7123/7 "2021-10-04T03:48:23Z")

</div>

There is nothing after line 27, but now I have another error regarding the logstash. Please refer to this error low below.

```auto
::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$ __VARARGS__ (/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:389)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
logstash | warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
logstash | LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
logstash | create at org/logstash/execution/ConvergeResultExt.java:135
logstash | add at org/logstash/execution/ConvergeResultExt.java:60
logstash | converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:402
logstash | [2021-10-04T03:42:06,313][ERROR][logstash.agent] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
logstash | [2021-10-04T03:42:06,333][FATAL][logstash.runner] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:402:in `block in converge_state'"]}
logstash | [2021-10-04T03:42:06,361][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
logstash | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash | at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.16.0.jar:?]
logstash | at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.16.0.jar:?]
logstash | at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:89) ~[?:?]

```

My logstash configuration is still the same as above.

Really appreciate any help.

---

<div class="post-metadata">

**Author:** ![searchymcsearchface](https://sea1.discourse-cdn.com/flex019/user_avatar/forum.opensearch.org/searchymcsearchface/32/868_2.png) [@searchymcsearchface](https://forum.opensearch.org/u/searchymcsearchface)\
**Post date:** [October 5, 2021, 8:32pm UTC](https://forum.opensearch.org/t/logstash-exited-with-code-1/7123/8 "2021-10-05T20:32:13Z")

</div>

It’s hard to say - I don’t see anything that stands out, to be honest. I would try to reduce your surface area to eliminate variable.

---

<div class="post-metadata">

**Author:** ![ritubajaj](https://avatars.discourse-cdn.com/v4/letter/r/9fc29f/32.png) [@ritubajaj](https://forum.opensearch.org/u/ritubajaj)\
**Post date:** [December 15, 2021, 7:53am UTC](https://forum.opensearch.org/t/logstash-exited-with-code-1/7123/9 "2021-12-15T07:53:40Z")

</div>

I have the same problem. I am trying to use logstash latest version 7.16.1 with Opendistro 1.13.3 to overcome log4j vulnerability. I understand we need to have opensearch output plugin to make them work.

I created a docker image as:  
FROM [docker.elastic.co/logstash/logstash:7.16.1](http://docker.elastic.co/logstash/logstash:7.16.1)  
RUN bin/logstash-plugin install logstash-output-opensearch

When applied via Helm charts, getting error:

[2021-12-15T07:35:45,752][ERROR][logstash.outputs.opensearch] Unknown setting ‘ilm\_enabled’ for opensearch  
[2021-12-15T07:35:45,835][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>“Java::JavaLang::IllegalStateException”, :message=\>“Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.”, :backtrace=\>[“org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)”, “org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)”, “org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)”, “org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)”, “org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)”, “org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)”, “org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)”, “usr.share.logstash.logstash\_minus\_core.lib.logstash.java\_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java\_pipeline.rb:47)”, “org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)”, “org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)”, “org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)”, “org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)”, “org.jruby.RubyClass.newInstance(RubyClass.java:939)”, “org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)”, “org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)”, “usr.share.logstash.logstash\_minus\_core.lib.logstash.pipeline\_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:52)”, “usr.share.logstash.logstash\_minus\_core.lib.logstash.pipeline\_action.create.RUBY$method$execute$0$ **VARARGS** (/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:50)”, “org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)”, “org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)”, “org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)”, “usr.share.logstash.logstash\_minus\_core.lib.logstash.agent.RUBY$block$converge\_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383)”, “org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)”, “org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)”, “org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)”, “org.jruby.runtime.Block.call(Block.java:139)”, “org.jruby.RubyProc.call(RubyProc.java:318)”, “org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)”, “java.base/java.lang.Thread.run(Thread.java:829)”]}  
warning: thread “Converge PipelineAction::Create” terminated with exception (report\_on\_exception is true):  
LogStash::Error: Don’t know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`  
create at org/logstash/execution/ConvergeResultExt.java:135  
add at org/logstash/execution/ConvergeResultExt.java:60  
converge\_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:396  
[2021-12-15T07:35:45,839][ERROR][logstash.agent] An exception happened when converging configuration {:exception=\>LogStash::Error, :message=\>“Don’t know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`”}  
[2021-12-15T07:35:45,846][FATAL][logstash.runner] An unexpected error occurred! {:error=\>#\<LogStash::Error: Don’t know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`\>, :backtrace=\>[“org/logstash/execution/ConvergeResultExt.java:135:in `create’”, “org/logstash/execution/ConvergeResultExt.java:60:in `add’”, “/usr/share/logstash/logstash-core/lib/logstash/agent.rb:396:in `block in converge\_state’”]}  
[2021-12-15T07:35:45,855][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit  
org.jruby.exceptions.SystemExit: (SystemExit) exit  
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]  
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]