What Clients, Agents, and Ingestion Tools do you use?

OpenDistro
1

Hi everyone, I wanted to learn more about the different tools people use with open source Elasticsearch, Open Distro for Elasticsearch, and OpenSearch. There are a lot of tools in this ecosystem and I want to learn more about what is popular. I would like to use this information to make recommendations on where OpenSearch should add testing, documentation, and other efforts for clients, agents, and ingestion tools.

NOTE: If there are any tools you are using that aren’t in this list, or if there is anything about one or more of these tools that you particularly like, please leave a comment with more details.

Thanks everyone!

Agents

tools that pickup and ship data from the host they are running on

Filebeat

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Metricbeat

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Packetbeat

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Winlogbeat

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Heartbeat

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Fluent Bit

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Fluentd

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Vector

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Open Telemetry Collector

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Ingestion Aggregators

tools that collect and transform data before shipping

Logstash

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Fluentd

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Open Telemetry Collector

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Data Streaming

tools that provide data pipelines and shipping to the cluster

Apache Kafka

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Apache Flink

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Apache Flume

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Clients

SDKs and Client Libraries

Open Telemetry libraries

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

B4J: jElasticsearch

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

C++: elasticlient

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Clojure: Spandex

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Clojure: Elastisch

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

ColdFusion: cbElasticSearch

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Erlang: erlastic_search

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Erlang: Tirexs

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Erlang: Elixir Bulk Processor

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Go: Official Elasticsearch Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Go: elastigo

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Go: elastic

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Go: elk

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Haskell: bloodhound

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Java: Low Level Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Java: Official High Level Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Java: Flummi

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Java: Jest

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

JavaScript: Official Elasticsearch Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Kotlin: ES Kotlin

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Kotlin: ES Kotlin Wrapper Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Lua: elasticsearch-lua

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

.Net: Official Elasticsearch Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Perl: Official Elasticsearch Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Perl: Elastijk

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

PHP: Official Elasticsearch Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

PHP: Elastica

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

PHP: elasticsearch

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

PHP: elasticsearcher

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Python: Official Elasticsearch Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

R: elastic

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

R: elasticdsl

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

R: uptasticsearch

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Ruby: Official Elasticsearch Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Ruby: elastics

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Ruby: chewy

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Ruby: Searchkick

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Ruby: Estella

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Rust: Official Elasticsearch Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Rust: rs-es

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Rust: elastic

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Scala: elastic4s

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Scala: escalar

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Scala: elasticsearch-client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Smalltalk: elasticsearch-smalltalk

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Swift: Elasticsearch NIO Client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Vert.x: elasticsearch-client

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

Spring Data Elasticsearch

  • I currently use
  • I don’t use, but I am interested in using
  • I don’t use, and I am not interested in using

0 voters

3

You missed… “I use, but wish I had a better alternative”

2 Likes
4

Somethings that may be useful to add…

  • Telegraf - Has an Elasticsearch output and is useful as an alternative to Metricbeat, especially for data sources not supported by Metricbeat or other components from Elastic.
  • Confluent Elasticsearch Connector for Kafka Connect - While Kafka is on the list, there are lots of ways to use Kafka without Kafka Connect. Personally I am not a fan of Connect and the Elasticsearch Connector, but it is somewhat widely used.
  • StreamSets and NiFi - I think of both of these as “Logstash with a GUI”. However like most GUIs, they are great for beginners, but eventually they just get in the way… at least IMO.
3 Likes
5

Good additions, we have customers using Telegraf, but we are using M3DB backend for time series :slight_smile:

1 Like
6

hm, i voted “yes” on Kafka - but we’re not using Kafka Connect, we have our own service in-between which consumes the Kafka messages from the topics we’re interested in, applies transformations and then writes this into ES. should i’ve voted “no”? if so, maybe rename “Kafka” to “Kafka Connect”?

1 Like
7

As additional client I would add the Hadoop/Apache Spark Elasticsearch connector: Elasticsearch for Apache Hadoop [8.4] | Elastic

1 Like
8

If you are using Kafka at all in your deployment topology to eventually write to ES or OpenSearch, yes is the way to vote. :slight_smile:

9

Thanks for sharing these details! I’ll note them down!

10

. Net seems to be missing the NEST library, the elasticsearch client is the lower level one (same as Java)

1 Like
11

Not sure if these two tools fit in with the rest, but we use ElastiFlow for netflow data and Wazuh as a SIEM

3 Likes
12

Closing the poll today. Thanks everyone for filling out the poll!