Logstash v8.1.1 compatibility with Opensearch

OpenSearch
1

Hi,

Logstash-OSS version 8.1.1 is been released few days ago and I am thinking to move to latest version.
Kindly update if logstash v8.1.1 is compatible with Opensearch v1.3.0.

Thanks in advance.

2

Hi @Pratiksha -

I think as long as your instance of logstash is using the logstash-output-plugin available from our github repository, it should work.

Hope that helps!

Nate

3

Hi @Pratiksha - I think I’m going to have to take back what I said. Logstash below 8.0 should be compatible with the logstash-output-opensearch, but the version that has been tested is 7.16.3. Above version 8 there are some compatibility issues.

4

Thanks @nateynate for the information. Can you please share by when support for higher version of logstash with opensearch can be expected.

5

Hi @nateynate
I have observed that even Logstash v7.17.x are not compatible with Opensearch. Can you please confirm whether it has the support or not. Is there any flag or workaround available for this?
Thanks in advance.

6

I’ve been running Logstash v7.17.x since its release. The only issue I’ve run into is that sometimes it forgets about logstash-output-opensearch, so when I restart the service I have reinstall the plugin.

7

It does appear that some other people have been successful with Logstash 7.17.x. When you say you’ve observed that they’re not compatible, do you have a particular log snippet you’d like to share? I’d love to see what you’re looking at.

Nate

8

Hi @nateynate
Logstash 7.17.x is compatible, I was earlier getting that error as I missed some configuration update.
Can you please share by when support for higher version of logstash with opensearch can be expected.
@reshippie Can you please share more details on what type of issue is it giving and what does it mean by forgetting logstash-output-opensearch plugin.

9

I haven’t had a chance to dig too deeply into the why of it, it may coincide with upgrades from 7.17.x to 7.17.y since those happen automatically on our systems, but very often when I restart Logstash the plugin is no longer in the registry. The files are still on disk, but the service doesn’t recognize this fact.

[2022-04-26T23:27:11,691][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"output", :name=>"opensearch"}
[2022-04-26T23:27:11,703][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:syslog, :exception=>"
Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'opensearch'. Are you
sure this is correct? Trying to load the opensearch output plugin resulted in this error: Unable to load the requested plugin named opensearch of type output. Th
e plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineEx
t.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initializ
e.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHel
pers.java:1169)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:84)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(I
nterpreterEngine.java:361)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)", "org.jruby.internal.runtime.methods
.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)", "org.jru
by.runtime.callsite.CachingCallSite.call(CachingCallSite.java:85)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInst
ance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logst
ash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minu
s_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "org.
jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.jav
a:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/us
r/share/logstash/logstash-core/lib/logstash/agent.rb:383)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.
IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.Rub
yProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}

Calling /usr/share/logstash/bin/logstash-plugin install logstash-output-opensearch resolves the issue, so it hasn’t been a high priority bug for me. If someone knows why or how to resolve, that would be wonderful.

1 Like
10

@reshippie This would be the perfect thing to file an issue in github for.

image

If you like, I can file the issue on your behalf, but you are otherwise perfectly welcome to file the issue as well. Make sure to precisely describe what version of the plugin you’re currently using, what the symptoms are, and especially the error itself as well as your workaround.

Nate