I am using opensearchproject/logstash-oss-with-opensearch-output-plugin:7.13.4 for logstash container. It cannot find the zip command.
Any suggestions?
when I tried this, logstash container stopped because, it could not find the output plugin ‘opensearch’.
what docker image are you using for logstash?
@usal - Hi, I am running the same setup as @blaklabz. I created a custom images with logstash 7.16.1 as base image and installing opensearch output plugin. Reference Dockerfile:
FROM docker.elastic.co/logstash/logstash:7.16.1
RUN bin/logstash-plugin install logstash-output-opensearch
And post that, I applied same changes as specified by @blaklabz i.e. replacing elasticsearch by opensearch.
Pointer : output configuration couldn’t contain ‘ilm_enabled’.
Has someone checked if the JndiLookup.class is used or bundled somewhere else in the framework(s)?
$ grep -ir "JndiLookup.class" /usr/
Binary file /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.3-java/vendor/jar-dependencies/org/logstash/inputs/logstash-input-tcp/6.0.3/logstash-input-tcp-6.0.3.jar matches
Should this be worrying that the class is in the logstash-input-tcp , or has this been detected already?
$ jar -tf logstash-input-tcp-6.0.3.jar | grep -ir JndiLookup
Binary file org/apache/logging/log4j/core/lookup/JndiLookup.class matches
Binary file org/apache/logging/log4j/core/lookup/Interpolator.class matches
Binary file META-INF/org/apache/logging/log4j/core/config/plugins/Log4j2Plugins.dat matches
Binary file logstash-input-tcp-6.0.3.jar matches
quote elastic.co on Apache Log4j2 Remote Code Execution (RCE) Vulnerability - CVE-2021-44228 - ESA-2021-31 - Security Announcements - Discuss the Elastic Stack
Logstash core will load the Log4j 2.15.0 from core instead of any other log4j-core jar in plugins, so no Jndi Lookups will happen.
However, mitigation can be done via:
zip -q -d /usr/share/logstash/vendor/bundle/jruby/*/gems/logstash-input-tcp-*-java/vendor/jar-dependencies/org/logstash/inputs/logstash-input-tcp/*/logstash-input-tcp-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
The “Logstash OSS with OpenSearch Output Plugin” on the download page and the corresponding Docker images are months old and are therefore are still using the vulnerable log4j version. Will there be an update? If anything there should be a big warning next to those downloads that they are insecure.
If using those packages is not recommended, they should be removed and replaced with instructions for how to use the original Logstash together with OpenSearch.
So I went to the link
https://hub.docker.com/r/opensearchproject/logstash-oss-with-opensearch-output-plugin
This was the link I used for the docker image
image:
repository: opensearchproject/logstash-oss-with-opensearch-output-plugin
tag: latest
But as others have mentioned, this image is 1 month old… Waiting on the update.
there is a new log4j 2.17.0 out.
Can a new OpenDistro docker image be published?
Would removing JndiLookup.class be enough to mitigate this vulnerability (CVE-2021-45105) which exists in 2.16? If not, then what is the mitigation process for this vulnerability both on OpenSearch and OpenDistro?
Also, if we remove old log4j jar files and replace them with 2.17.0 jars would that be OK and supported both for OpenSearch and OpenDistro?
In my personal testing that works just fine, in the sense that I’ve done this with a development cluster that I individually use, the cluster starts up and behaves as expected. The Log4J2 team is very attentive to backward compatibility. However, this isn’t a configuration that the full suite of tests has been run against, which is what I would need to endorse it as a solution.
The OpenSearch project team will publish tested versions of OpenSearch that include Log4J2 2.17.0 shortly, and a new docker image that includes Logstash 7.16.2 (which itself contains Log4J2 2.17.0).
If you’re using Open Distro, I’d strongly recommend this as the right time to upgrade to OpenSearch. A lot of work has gone into making the upgrade process smooth.
1 Like
May I bump this question up? For those who can’t upgrade immediately, is there any other action we need to mitigate CVE-2021-45105? I didn’t see any calls like ${ctx:<some_variable>} in the OpenSearch code base but maybe I’m looking at this too simply.
The only mitigation to CVE-2021-45105 described by the Log4j maintainers is to update to Log4j 2.17.0. OpenSearch version 1.2.3 was released this morning and updates the Log4j version, so I’d suggest upgrading to that.
1 Like
Hi JulesGraybill, I was looking at the mitigation described here by Apache regarding CVE-2021-45105:
Upgrade to Log4j 2.3.1 (for Java 6), 2.12.3 (for Java 7), or 2.17.0 (for Java 8 and later).
Alternatively, this infinite recursion issue can be mitigated in configuration:
In PatternLayout in the logging configuration, replace Context Lookups like ${ctx:loginId} or $${ctx:loginId} with Thread Context Map patterns (%X, %mdc, or %MDC). Otherwise, in the configuration, remove references to Context Lookups like ${ctx:loginId} or $${ctx:loginId} where they originate from sources external to the application such as HTTP headers or user input.
However, I didn’t find any explicit Pattern Layouts with that format in the code base. Are those 2 items above applicable to the OpenSearch code base as it is right now?
Hi @silver_searcher, you’re right that OpenSearch’s codebase doesn’t use pattern layouts in the format that could cause the DOS described in the CVE. So in OpenSearch’s out-of-the-box configuration, we haven’t reproduced the DOS.
Informally, I see no reason you wouldn’t be able to mitigate this CVE by avoiding adding a known-bad configuration to a cluster running OpenSearch < 1.2.3. The remediation we’re recommending as a project is for users to upgrade, because that provides the most comprehensive and provable protection in all configurations, and we’ve heard from many users it’s a priority for them to be running no software that contains Log4j2 < 2.17.0.
1 Like
Hi @searchymcsearchface and @JulesGraybill
We use 3 standalone OpenDistro plugins (security, alerting, job scheduling). Do we need to update the OpenDistro security plugin as we can see the log4j-slf4j-impl-2.11.1.jar there?
I could not find a github release for 1.13.3 for the security plugin, can you help me out with the update process?(if an update is required)
Thanks in advance
Hi @nileshgajwani,
The way that CVE-2021-44228 was addressed in 1.13.3 was by removing the JndiLookup class from the Log4j classpath (see https://opendistro.github.io/for-elasticsearch/blog/2021/12/update-to-1-13-3/). There was no newer version released for the security plugin, so 1.13.1 is still the latest one. There is no update indicated specifically for log4j-slf4j-impl-2.11.1.jar, so version 1.13.1 of the security plugin (and others that also didn’t get new releases to address this CVE) are the correct versions provided you are running them with the 1.13.3 version of OpenDistro.
Hope that helps clarify things!
2 Likes