Log4j2 vulnerability in OpenSearch

Hey folks - OpenSearch and Open Distro have both been updated.

3 Likes

Do we know if there is likely to be any backports available for the earlier versions of OpenDistro too?

Hello @daleo - welcome to the forums.

The only release update I know of for OpenDistro is 1.13.3.
This blog post has a link to further information regarding “For those who cannot upgrade to 1.13.3”:
https://opendistro.github.io/for-elasticsearch/blog/2021/12/update-to-1-13-3/

Checking to make sure - have all the repositories been updated as well? Yum, Apt, Dockerhub etc…

Checking to make sure - have all the repositories been updated as well? Yum, Apt, Dockerhub etc…

i’m wondering the same… we’re working on upgrading to a docker based opensearch system but still have debian package based open distro cluster out there… could really use a open distro debian package release for this vuln

1 Like

The Open Distro deb and rpm packages are built on top of the upstream Elasticsearch deb and rpm packages, and those upstream packages are what contains the Log4j2 jars. So we’ve been looking really hard at what the options are for distributing a release that contains a fix, and there are fewer options here than there are for the .tgz and Docker builds.

Right now it looks like we’d need to develop a deb & rpm packaging process more or less from scratch, and doing that in a way that retains backward compatibility and upgradeability for existing users is underway and will likely take several weeks of development.

If you’re using Open Distro < 1.13.2 or the Open Distro 1.13.2 deb/rpm packages, please apply one of the mitigations from the Log4j2 website in the section beginning “For those who cannot upgrade to 2.15.0…”. All users can and should do this immediately, while we work to create deb/rpm packages that have at least one of these mitigations built-in by default.

2 Likes

HI @daleo , Open Distro has historically always applied security fixes to the latest version. If you are unable for any reason to upgrade to version 1.13.3, please apply one of the mitigations from the Log4j2 website in the section beginning “For those who cannot upgrade to 2.15.0…”.

2 Likes

Is there any plan to release stand-alone OpenDistro plugins for 1.13.3?
(Like described here: Standalone Elasticsearch Plugin Install - Open Distro Documentation)

Hi,
Which logstash do you recomand to use for opendistro 1.13.3 in order to have that vulns fixed?
They are gonna fix for logstash 7.16.1 which isn’t compatible with opendistro.
Thanks!

Any updates for those of us who are using helm to deploy opendistro? I upgraded the image to 1.13.3 and got an error.

#editz - so looks like if i leave the kibana image to 1.13.2 and change the elasticsearch image to 1.13.3 we are good the service starts. Can I get confirmation this is the desired state for helm?

Ahh thanks for that link, this confirms what i was seeing.

Let me verify what’s being done on that front. As I understanding it, the Open Distro distribution was patched by removing class paths, not really a rebuild.

Humm. opendistro-build#792 was merged for 1.13.3 regarding helm. Could you throw an issue on that repo describing the error you got?

Logstash 7.16.1 should work fine with Open Distro, as long as you use the OpenSearch output plugin. We’re pushing a new version of the Logstash+OpenSearch-output-plugin distribution to the downloads page shortly, but you can also upgrade or mitigate self-service by pulling the new version of Logstash right now directly from its download page, or by following the mitigation instructions provided for Logstash last week.

1 Like

@daleo out of curiosity, what version of Open Distro are you using? I’d love to understand what help could make it easy for you to upgrade to OpenSearch or Open Distro 1.13.3.

1 Like

Hi,

for those who can’t upgrade on short term, because of too many dependencies, please find below my notes and walkthrough to get this mitigated for OpenDistro <1.13.2

The most important mitigation is for Logstash (if used)!

LOG4J CVE-2021-44228

Vulnerability Indicators

  • Log4j version – all 2.x versions before 2.15.0 (released today, Friday, December 10, 2021) are affected

    • Affected: all versions from 2.0-beta9 to 2.14.1
  • Log4j version v1.xcve-details
    if the ‘log4j.properties’ file contains a ‘JMSAppender’ config line (find / -name "log4j.properties" |xargs grep -i JMSAppend)

  • JVM version - if lower than:

    • Java 6 – 6u212
    • Java 7 – 7u202
    • Java 8 – 8u192
    • Java 11 - 11.0.2

Apache - Manual deactivation/mitigation of Log4j’s JNDI:

https://logging.apache.org/log4j/2.x/

For those who cannot upgrade to 2.15.0, in releases >=2.10, this behavior can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. For releases >=2.7 and <=2.14.1, all PatternLayout patterns can be modified to specify the message converter as %m{nolookups} instead of just %m. For releases >=2.0-beta9 and <=2.10.0, the mitigation is to remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.

Amazon OpenDistro and Opensearch patches and infos:

https://opendistro.github.io/for-elasticsearch/blog/2021/12/update-to-1-13-3/

Elastic

Manual mitigation diary and walkthrough

First gathering some information.
In this example on a CentOS 7.x server.

log4j

Version: 2.11.1

Where is it , and how to find it:

find /usr/ -type f -name "log4j-core*"
/usr/share/elasticsearch/lib/log4j-core-2.11.1.jar
/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar

java

Version: 11.0.7

rpm -qa | grep openjdk-devel
java-11-openjdk-devel-11.0.7.10-4.el7_8.x86_64
java --version
openjdk 11.0.7 2020-04-14 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.7+10-LTS)

OpenDistro

Version: 1.2.1

rpm -qa | grep opendistroforelasticsearch
opendistroforelasticsearch-1.2.1-1.noarch

Elasticsearch

Version: 7.2.1

rpm -qa | grep elasticsearch
elasticsearch-oss-7.2.1-1.x86_64

Logstash

Version: 7.4.2-1

rpm -qa | grep logstash
logstash-oss-7.4.2-1.noarch

Mitigation for Logstash

Mitigation requires removal of the JndiLookup Class or update to Logstash version 6.8.21 or 7.16.1

1.) via jvm.option

vim /etc/logstash/jvm.options

add following lines:

# CVE-2021-44228 mitigation
-Dlog4j2.formatMsgNoLookups=true

Restart logstash and check if option is active via ps -axfww| grep -i log4j

/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom `-Dlog4j2.isThreadContextMapInheritable=true` -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.9.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.24.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:`/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.11.1.jar`:`/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar`:`/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.11.1.jar`:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash

2.) remove JndiLookup.class from log4j-core-2.* as logstash should not need it

zip -q -d /usr/share/logstash/logstash-core/lib/jars/log4j-core-2.* org/apache/logging/log4j/core/lookup/JndiLookup.class

Restart logstash.

Mitigation for Elasticsearch

For Elasticsearch 5.6.11+, 6.4+, and 7.0+ setting proper JVM options provides full protection against the RCE and information leak attacks.

1.) via jvm.option

vim /etc/logstash/jvm.options

add following lines:

# CVE-2021-44228 mitigation
-Dlog4j2.formatMsgNoLookups=true

Restart elasticsearch and check if option is active via ps -axfww| grep -i log4j

2 Likes

Hello!

Please, does opendistro-performance-analyzer plugin needs any mitigation or only add the mitigation to jvm.options on /etc/elasticsearch is enough?

Thanks in advance.

Hey Jules I’m running into an issue where OSS-Logstash 7.16.1 is saying that the 1.13.3 is not compatible and then exits. Has anyone run into this?

[2021-12-14T17:40:12,925][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::Configurati
onError: Could not connect to a compatible version of Elasticsearch>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logs
tash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:247:in `block in healthcheck!'", "org/jruby/Ru
byHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/out
puts/elasticsearch/http_client/pool.rb:240:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elastics
earch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:374:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.
5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:89:in `update_initial_urls'", "/u
sr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/p
ool.rb:83:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/e
lasticsearch/http_client.rb:359:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-
java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-ou
tput-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client'", "/usr/share/logstash
/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in
`build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/plugin_mixins/elastics
earch/common.rb:34:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/lo
gstash/outputs/elasticsearch.rb:275:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logsta
sh/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:2
32:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipelin
e.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins'", "/usr/
share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pi
peline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["
/usr/share/logstash/pipeline/input_main", "/usr/share/logstash/pipeline/output_main"], :thread=>"#<Thread:0x30bc7e67 run>"}
[2021-12-14T17:40:12,961][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2021-12-14T17:40:13,022][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::F
ailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2021-12-14T17:40:13,220][INFO ][logstash.runner          ] Logstash shut down.
[2021-12-14T17:40:13,243][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

Hello,
are you using the logstash opensearch output plugin?
You can download logstash + opensearch output plugin bundle from Opensearch 2.2.1 · OpenSearch

Hi @blaklabz, yeah, that connection error is actually in the elasticsearch-output plugin - which is looking for a range of matching Elasticsearch version numbers that doesn’t include OpenSearch or any of the open source Elasticsearch builds (such as the one included in Open Distro.) You can get the opensearch-output plugin from rubygems that will run on Logstash 7.16.1 and connect to an OpenSearch or Open Distro cluster. We’ll also have packages on the OpenSearch downloads page shortly that include both of these things bundled together for convenience.

Hello, are the mitigations for ElasticSearch listed here still valid now since a new log4j 2.16 version has been released and previous mitigations having been partially discredited?