Log4j2 vulnerability in OpenSearch

Security
, ,
23

Humm. opendistro-build#792 was merged for 1.13.3 regarding helm. Could you throw an issue on that repo describing the error you got?

24

Logstash 7.16.1 should work fine with Open Distro, as long as you use the OpenSearch output plugin. We’re pushing a new version of the Logstash+OpenSearch-output-plugin distribution to the downloads page shortly, but you can also upgrade or mitigate self-service by pulling the new version of Logstash right now directly from its download page, or by following the mitigation instructions provided for Logstash last week.

1 Like
25

@daleo out of curiosity, what version of Open Distro are you using? I’d love to understand what help could make it easy for you to upgrade to OpenSearch or Open Distro 1.13.3.

1 Like
26

Hi,

for those who can’t upgrade on short term, because of too many dependencies, please find below my notes and walkthrough to get this mitigated for OpenDistro <1.13.2

The most important mitigation is for Logstash (if used)!

LOG4J CVE-2021-44228

Vulnerability Indicators

  • Log4j version – all 2.x versions before 2.15.0 (released today, Friday, December 10, 2021) are affected

    • Affected: all versions from 2.0-beta9 to 2.14.1

  • Log4j version v1.x → cve-details
    if the ‘log4j.properties’ file contains a ‘JMSAppender’ config line (find / -name "log4j.properties" |xargs grep -i JMSAppend)

  • JVM version - if lower than:

    • Java 6 – 6u212
    • Java 7 – 7u202
    • Java 8 – 8u192
    • Java 11 - 11.0.2

Apache - Manual deactivation/mitigation of Log4j’s JNDI:

https://logging.apache.org/log4j/2.x/

For those who cannot upgrade to 2.15.0, in releases >=2.10, this behavior can be mitigated by setting either the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to true. For releases >=2.7 and <=2.14.1, all PatternLayout patterns can be modified to specify the message converter as %m{nolookups} instead of just %m. For releases >=2.0-beta9 and <=2.10.0, the mitigation is to remove the JndiLookup class from the classpath: zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class.

Amazon OpenDistro and Opensearch patches and infos:

https://opendistro.github.io/for-elasticsearch/blog/2021/12/update-to-1-13-3/

Elastic

Manual mitigation diary and walkthrough

First gathering some information.
In this example on a CentOS 7.x server.

log4j

Version: 2.11.1

Where is it , and how to find it:

find /usr/ -type f -name "log4j-core*"

/usr/share/elasticsearch/lib/log4j-core-2.11.1.jar
/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar

java

Version: 11.0.7

rpm -qa | grep openjdk-devel
java-11-openjdk-devel-11.0.7.10-4.el7_8.x86_64

java --version
openjdk 11.0.7 2020-04-14 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.7+10-LTS)

OpenDistro

Version: 1.2.1

rpm -qa | grep opendistroforelasticsearch
opendistroforelasticsearch-1.2.1-1.noarch

Elasticsearch

Version: 7.2.1

rpm -qa | grep elasticsearch
elasticsearch-oss-7.2.1-1.x86_64

Logstash

Version: 7.4.2-1

rpm -qa | grep logstash
logstash-oss-7.4.2-1.noarch

Mitigation for Logstash

Mitigation requires removal of the JndiLookup Class or update to Logstash version 6.8.21 or 7.16.1

1.) via jvm.option

vim /etc/logstash/jvm.options

add following lines:

# CVE-2021-44228 mitigation
-Dlog4j2.formatMsgNoLookups=true

Restart logstash and check if option is active via ps -axfww| grep -i log4j

/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom `-Dlog4j2.isThreadContextMapInheritable=true` -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.9.3.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.9.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.11.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.24.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:`/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.11.1.jar`:`/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar`:`/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.11.1.jar`:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash

2.) remove JndiLookup.class from log4j-core-2.* as logstash should not need it

zip -q -d /usr/share/logstash/logstash-core/lib/jars/log4j-core-2.* org/apache/logging/log4j/core/lookup/JndiLookup.class

Restart logstash.

Mitigation for Elasticsearch

For Elasticsearch 5.6.11+, 6.4+, and 7.0+ setting proper JVM options provides full protection against the RCE and information leak attacks.

1.) via jvm.option

vim /etc/logstash/jvm.options

add following lines:

# CVE-2021-44228 mitigation
-Dlog4j2.formatMsgNoLookups=true

Restart elasticsearch and check if option is active via ps -axfww| grep -i log4j

2 Likes
27

Hello!

Please, does opendistro-performance-analyzer plugin needs any mitigation or only add the mitigation to jvm.options on /etc/elasticsearch is enough?

Thanks in advance.

28

Hey Jules I’m running into an issue where OSS-Logstash 7.16.1 is saying that the 1.13.3 is not compatible and then exits. Has anyone run into this?

[2021-12-14T17:40:12,925][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::Configurati
onError: Could not connect to a compatible version of Elasticsearch>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logs
tash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:247:in `block in healthcheck!'", "org/jruby/Ru
byHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/out
puts/elasticsearch/http_client/pool.rb:240:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elastics
earch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:374:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.
5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:89:in `update_initial_urls'", "/u
sr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/p
ool.rb:83:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/e
lasticsearch/http_client.rb:359:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-
java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-ou
tput-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client'", "/usr/share/logstash
/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in
`build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/plugin_mixins/elastics
earch/common.rb:34:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/lo
gstash/outputs/elasticsearch.rb:275:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logsta
sh/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:2
32:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipelin
e.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins'", "/usr/
share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pi
peline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["
/usr/share/logstash/pipeline/input_main", "/usr/share/logstash/pipeline/output_main"], :thread=>"#<Thread:0x30bc7e67 run>"}
[2021-12-14T17:40:12,961][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2021-12-14T17:40:13,022][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::F
ailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2021-12-14T17:40:13,220][INFO ][logstash.runner          ] Logstash shut down.
[2021-12-14T17:40:13,243][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
29

Hello,
are you using the logstash opensearch output plugin?
You can download logstash + opensearch output plugin bundle from Opensearch 2.2.1 · OpenSearch

30

Hi @blaklabz, yeah, that connection error is actually in the elasticsearch-output plugin - which is looking for a range of matching Elasticsearch version numbers that doesn’t include OpenSearch or any of the open source Elasticsearch builds (such as the one included in Open Distro.) You can get the opensearch-output plugin from rubygems that will run on Logstash 7.16.1 and connect to an OpenSearch or Open Distro cluster. We’ll also have packages on the OpenSearch downloads page shortly that include both of these things bundled together for convenience.

31

Hello, are the mitigations for ElasticSearch listed here still valid now since a new log4j 2.16 version has been released and previous mitigations having been partially discredited?

32

Hi, Has anyone installed opensearch-output-plugin with logstash 7.16.1 to make it work with Opendistro 1.13.3?

I am getting below error:
[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>“Java::JavaLang::IllegalStateException”, :message=>“Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.”, :backtrace=>[“org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:119)”, “org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)”

Not sure if this plugin will require change in config file ? Do we have any clear documentation where it states that opensearch plugin can send data to elasticsearch (opendistro) ?

33

Not only logstash, but also bundled plugins can come with parts of the log4j library, however the vulnerable part is in the log4j-core-*.jar (<2.16.0), so they should be safe and NOT be affected.

Makes you wonder where it’s all being used. Let’s hope that these libraries will soon be subject to a security review/audit.

$ find /usr/share/ -type f -name "log4j*.jar"
/usr/share/elasticsearch/lib/log4j-1.2-api-2.11.1.jar
/usr/share/elasticsearch/lib/log4j-api-2.11.1.jar
/usr/share/elasticsearch/lib/log4j-core-2.11.1.jar
/usr/share/elasticsearch/plugins/opendistro_security/log4j-slf4j-impl-2.11.1.jar
/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.11.1.jar
/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.11.1.jar
/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.11.1.jar
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http-3.3.0-java/vendor/jar-dependencies/org/apache/logging/log4j/log4j-api/2.11.1/log4j-api-2.11.1.jar
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_event_hubs-1.1.2/vendor/jar-dependencies/org/apache/logging/log4j/log4j-api/2.9.1/log4j-api-2.9.1.jar
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_event_hubs-1.1.2/vendor/jar-dependencies/org/apache/logging/log4j/log4j-slf4j-impl/2.9.1/log4j-slf4j-impl-2.9.1.jar
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.3-java/vendor/jar-dependencies/org/apache/logging/log4j/log4j-api/2.11.1/log4j-api-2.11.1.jar

For all those who like to know what frameworks are potentially affected, use following find command:

find /usr/ -type f -name "log4j-core*

To find running processes that actively using vulnerable part of log4j:

ps -axfww | grep -i log4j-core

It’s not an indication to be 100% safe, if the commands above don’t trigger, but at least a start.

34

Hi,

I manually updated (just) the log4j to 2.16.0 for/in logstash-oss7.4.2, please find the steps below.

Disclaimer:

  • before using it, take any chance to update logstash via original package
  • use at your own risk!
  • test before use in production!
  • this is not an official guide from OpenDistro/OpenSearch
  • is it perfect → no → consider it as frankenstein solution, just slightly better than mitigating
  • it might cause yet unknown packaging/dependencies issues

Steps

# stop logstash
systemctl stop logstash

# download log4j from apache side
wget https://www.apache.org/dyn/closer.lua/logging/log4j/2.16.0/apache-log4j-2.16.0-bin.tar.gz

# uncompress
tar -xzvf apache-log4j-2.16.0-bin.tar.gz

# backup original logstash log4j libraries
mkdir bak
mv -v /usr/share/logstash/logstash-core/lib/jars/log4j* bak/

# deploying new log4j version 2.16.0
cp -av apache-log4j-2.16.0-bin/log4j-api-2.16.0.jar /usr/share/logstash/logstash-core/lib/jars/
cp -av apache-log4j-2.16.0-bin/log4j-core-2.16.0.jar /usr/share/logstash/logstash-core/lib/jars/
cp -av apache-log4j-2.16.0-bin/log4j-slf4j-impl-2.16.0.jar /usr/share/logstash/logstash-core/lib/jars/

# correct ownership of log4j libraries
chown -v logstash:logstash /usr/share/logstash/logstash-core/lib/jars/log4j*

# start logstash
systemctl start logstash
1 Like
36

ad. opendistro-performance-analyzer plugin → This is indeed a very good question!

I haven’t tested it, and it might not be vulnerable at all, but just in case, and as it runs separately and without considering jvm.options from elasticsearch , I’d tend to add the mitigation option -Dlog4j2.formatMsgNoLookups=true in the start bash script which is used to start performance analyzer (,… just in case).

@searchymcsearchface, I’d assume this has been discussed already, what’s the experts opinion on that?

In my installation the performance analyzer start script is located here:

/usr/share/elasticsearch/plugins/opendistro_performance_analyzer/pa_bin/performance-analyzer-agent

Add mitigation options -Dlog4j2.formatMsgNoLookups=true right next to the other log4j option like below should be sufficient:

#...

if ! echo $* | grep -E '(^-d |-d$| -d |--daemonize$|--daemonize )' > /dev/null; then
    exec $JAVA_HOME/bin/java -Des.path.home=$ES_HOME -Dlog4j.configurationFile=$ES_HOME/plugins/opendistro_performance_analyzer/pa_config/log4j2.xml -Dlog4j2.formatMsgNoLookups=true\
    -DconfigFilePath=$3 \
    -Xms64M -Xmx64M -XX:+UseSerialGC -XX:CICompilerCount=1 -XX:-TieredCompilation -XX:InitialCodeCacheSize=4096 \
    -XX:InitialBootClassLoaderMetaspaceSize=30720 -XX:MaxRAM=400m \
    -cp $ES_HOME/lib/*:$ES_HOME/plugins/opendistro_performance_analyzer/* com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp
else
    echo 'Starting deamon'
    exec $JAVA_HOME/bin/java -Des.path.home=$ES_HOME -Dlog4j.configurationFile=$ES_HOME/plugins/opendistro_performance_analyzer/pa_config/log4j2.xml -Dlog4j2.formatMsgNoLookups=true\
        -DconfigFilePath=$3 \
        -Xms64M -Xmx64M -XX:+UseSerialGC -XX:CICompilerCount=1 -XX:-TieredCompilation -XX:InitialCodeCacheSize=4096 \
        -XX:InitialBootClassLoaderMetaspaceSize=30720 -XX:MaxRAM=400m \
        -cp $ES_HOME/lib/*:$ES_HOME/plugins/opendistro_performance_analyzer/* com.amazon.opendistro.elasticsearch.performanceanalyzer.PerformanceAnalyzerApp &

#...
37

The current advised mitigation (I think/as of writing/etc.) is to remove the classpath entirely - I don’t think the flag is sufficient. My understanding is that anything in the updated Open Distro bundle is covered with the 1.13.3. Lemme double check.

1 Like
39

@searchymcsearchface @JulesGraybill

Okay, looks like I was able to use the docker image in the helm chart. The service actually started this time, but it’s still giving me an error about license…

[2021-12-15T15:55:48,720][ERROR][logstash.outputs.elasticsearch][main] Unable to get license information {:url=>"https://admin:xxxxxx@opendistro-opendi
stro-es-client-service:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :message=>"Got response code '400'
 contacting Elasticsearch at URL 'https://opendistro-opendistro-es-client-service:9200/_license'"}
[2021-12-15T15:55:48,721][ERROR][logstash.outputs.elasticsearch][main] Could not connect to a compatible version of Elasticsearch {:url=>"https://admin
:xxxxxx@opendistro-opendistro-es-client-service:9200/"}
40

Are there any more details on the OpenDistro 1.13.3 security fix?

I got that you removed JndiLookup class from the Log4j classpath, but where can I see the commit/change in sources for this?

41

Sure. The team has unpackaged, removed JndiLookup.class and packaged it back. The sources are not modified hence no commits.

You can validate the changes via:

  1. Opendistro versions < 1.13.3
:~/opendistroforelasticsearch-1.13.2$ grep -r "org/apache/logging/log4j/core/lookup/JndiLookup.class" . 
Binary file ./lib/log4j-core-2.11.1.jar matches 
Binary file ./plugins/opendistro-performance-analyzer/performance-analyzer-rca/lib/log4j-core-2.13.0.jar matches 
Binary file ./performance-analyzer-rca/lib/log4j-core-2.13.0.jar matches
  1. Opendistro 1.13.3
:~/opendistroforelasticsearch-1.13.3$ grep -r "org/apache/logging/log4j/core/lookup/JndiLookup.class" .  
:~/opendistroforelasticsearch-1.13.3$
1 Like
42

Thank you for confirming the docker image came from OpenDistro team.

Having docker hub image without github commit and release, is very suspect to supply chain attack. If possible, would be good to add the steps in a Dockerfile and push another version, so we have github and docker matching releases.

43

Thanks for the details. But those are from the source tarball. As I am relying on Docker images and as dburdick mentionned, I would like a proof for the Docker images content. Currently it looks like this was done by hand, and we have no way of knowing what has changed within the image.

Adding full command for the sake of completeness

$ curl -s https://d3g5vo6xdbdb9a.cloudfront.net/tarball/opendistro-elasticsearch/opendistroforelasticsearch-1.13.2-linux-x64.tar.gz | tar xz
$ grep -r "org/apache/logging/log4j/core/lookup/JndiLookup.class" opendistroforelasticsearch-1.13.2
grep: opendistroforelasticsearch-1.13.2/performance-analyzer-rca/lib/log4j-core-2.13.0.jar: binary file matches
grep: opendistroforelasticsearch-1.13.2/lib/log4j-core-2.11.1.jar: binary file matches
grep: opendistroforelasticsearch-1.13.2/plugins/opendistro-performance-analyzer/performance-analyzer-rca/lib/log4j-core-2.13.0.jar: binary file matches

$ curl -s https://d3g5vo6xdbdb9a.cloudfront.net/tarball/opendistro-elasticsearch/opendistroforelasticsearch-1.13.3-linux-x64.tar.gz | tar xz
$ grep -r "org/apache/logging/log4j/core/lookup/JndiLookup.class" opendistroforelasticsearch-1.13.3
44

Okay, for those of you who are on the same boat as me…

I had the ELK stack of opendistro 1.13.2 paired with logstash-OSS and filebeat-OSS 7.10.2. This worked fine for me although the security configuration of this thing makes me cry because we run everything in rancher (k3s)

The log4j bomb dropped, there was a scramble to move away from anything that could be an issue. This is when I upgraded the opendistro to 1.13.2 kibana and 1.13.3 elastic. I then scrambled to use 7.16.1 for both logstash and filebeat. This is when I began running into issues… namely the version missmatch.

After some digging, it seemed that my chart was correct, the only thing, it still wouldn’t work. Turns out that I had to make one simple change. And I figured I would put that here.

I had to change my output configuration from elasticsearch to opensearch.

   opensearch {
        hosts => ["https://opendistro-opendistro-es-client-service:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
        ssl => true
        ssl_certificate_verification => false
        user => admin
        password => admin
        cacert => '/etc/certs/root-ca.pem'
       }
    }

instead of…

   elasticsearch {
        hosts => ["https://opendistro-opendistro-es-client-service:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
        ssl => true
        ssl_certificate_verification => false
        user => admin
        password => admin
        cacert => '/etc/certs/root-ca.pem'
       }
    }
2 Likes