Logstash-oss with non-removable x-pack

Open Source Elasticsearch and Kibana
1

Hi,

anyone else experienced that the logstash-oss version from the original elastic site contain x-pack stuff which to my understanding should not be in the oss packages?

  • install oss version from website (sidenote, oss doesn’t seem to be in the repository)
    curl https://artifacts.elastic.co/downloads/logstash/logstash-oss-6.7.0.rpm -o logstash-oss-6.7.0.rpm

  • when starting the oss version following error pops up:
    May 02 11:53:05 host logstash[8117]: [ERROR] 2019-05-02 11:53:05.366 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) no such file to load -- x-pack/logstash_registry

  • ok, then I thought, just let’s remove logstash x-pack plugin
    /usr/share/logstash/bin/logstash-plugin remove x-pack
    → nope :thinking:
    WARNING: All illegal access operations will be denied in a future release wAs of 6.3+ X-Pack is part of the default distribution and cannot be uninstalled; to remove all X-Pack features, you must install the OSS-only distribution of Logstash. ERROR: Operation aborted, cannot remove plugin, message: x-pack not a removable plugin

  • it was not working either way, so I had to remove the x-pack plugin from ugly way:
    mv -v /usr/share/logstash/x-pack /tmp/

Maybe I have overlooked something or maybe elastic doesn’t have a highest interest in creating functional oss packages?

thanks,
nean

2

Update:
i created an issue at github:

also I found another workaround:
one can deactivate/comment load_xpack unless LogStash::OSS in the code/file /usr/share/logstash/logstash-core/lib/logstash/plugins/registry.rb like below

def setup!
  #load_xpack unless LogStash::OSS
  load_available_plugins
  execute_universal_plugins
end
3

Update:
Elastic applied patches → logstash-oss not starting and contain non-removable x-pack · Issue #10783 · elastic/logstash · GitHub ,
However, fully fixed OSS packages will be available only as of version 6.8.1 and 7.2 , unfortunately those versions are quite ahead and not yet supported by ODfE.

Currently, with ODfE 1.0.0 and Elasticsearch/Logstash 7.0.1 one has to manually apply the hacks to get rid of the issues caused by the x-pack code fragments.
In addition ilm - index lifecycle management is exclusively for x_pack (a x-pack feature) and causing issues on 7.0.1, so its recommended to disable it in the logstash config:

output {
  elasticsearch { 
    hosts =>  [.. ]
    ilm_enabled => false
  }
}

As I could figure out, when using OSS packages (also >7.2), there is still x-pack related code in there, which probably slowly, or in worst case, never, get removed and might cause issues in the future.

Question to ODfE:
What is the strategy to limit/get rid of issues caused by x-pack code in OSS packages?

thanks,
nean

4

@nean Thanks for you post. this has helped me to resolve my issue with logstash output to elasticsearch.

Thanks,
Ashwin

5

Elastic did it again, the Logstash OSS version is not OpenSource and still requires proprietary “x-pack” components to run.

# yum info logstash-oss
Loaded plugins: langpacks, rhnplugin, versionlock
This system is receiving updates from RHN Classic or Red Hat Satellite.
Installed Packages
Name        : logstash-oss
Arch        : noarch
Epoch       : 1
Version     : 7.4.2
Release     : 1
Size        : 289 M
Repo        : installed
From repo   : elasticsearch-7.x
Summary     : An extensible logging pipeline
URL         : http://www.elasticsearch.org/overview/logstash/
License     : ASL 2.0
Description : An extensible logging pipeline

This does not start:

Nov 07 19:08:39 logstash[249857]: [ERROR] 2019-11-07 19:08:39.737 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) no such file to load -- monitoring/monitoring

# grep -R "monitoring/monitoring" /usr/share/logstash/*
/usr/share/logstash/x-pack/lib/x-pack/logstash_registry.rb:require "monitoring/monitoring"

EDIT: it seems I got it running after I did a manual cleanup

yum install logstash
rm -rvf/usr/share/logstash/
yum install logstash-oss
6

@nean thanks for keeping us posted!