What Clients, Agents, and Ingestion Tools do you use?

elifish · June 18, 2021, 6:39pm

Hi everyone, I wanted to learn more about the different tools people use with open source Elasticsearch, Open Distro for Elasticsearch, and OpenSearch. There are a lot of tools in this ecosystem and I want to learn more about what is popular. I would like to use this information to make recommendations on where OpenSearch should add testing, documentation, and other efforts for clients, agents, and ingestion tools.

NOTE: If there are any tools you are using that aren’t in this list, or if there is anything about one or more of these tools that you particularly like, please leave a comment with more details.

Thanks everyone!

Agents

tools that pickup and ship data from the host they are running on

Filebeat

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Metricbeat

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Packetbeat

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Winlogbeat

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Heartbeat

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Fluent Bit

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Fluentd

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Vector

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Open Telemetry Collector

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Ingestion Aggregators

tools that collect and transform data before shipping

Logstash

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Fluentd

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Open Telemetry Collector

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Data Streaming

tools that provide data pipelines and shipping to the cluster

Apache Kafka

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Apache Flink

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Apache Flume

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Clients

SDKs and Client Libraries

Open Telemetry libraries
[poll name= otellib results=always min=1 max=1 chartType=bar]

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using
[/poll]

B4J: jElasticsearch

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

C++: elasticlient

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Clojure: Spandex

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Clojure: Elastisch

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

ColdFusion: cbElasticSearch

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Erlang: erlastic_search

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Erlang: Tirexs

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Erlang: Elixir Bulk Processor

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Go: Official Elasticsearch Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Go: elastigo

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Go: elastic

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Go: elk

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Haskell: bloodhound

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Java: Low Level Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Java: Official High Level Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Java: Flummi

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Java: Jest

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

JavaScript: Official Elasticsearch Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Kotlin: ES Kotlin

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Kotlin: ES Kotlin Wrapper Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Lua: elasticsearch-lua

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

.Net: Official Elasticsearch Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Perl: Official Elasticsearch Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Perl: Elastijk

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

PHP: Official Elasticsearch Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

PHP: Elastica

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

PHP: elasticsearch

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

PHP: elasticsearcher

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Python: Official Elasticsearch Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

R: elastic

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

R: elasticdsl

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

R: uptasticsearch

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Ruby: Official Elasticsearch Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Ruby: elastics

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Ruby: chewy

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Ruby: Searchkick

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Ruby: Estella

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Rust: Official Elasticsearch Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Rust: rs-es

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Rust: elastic

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Scala: elastic4s

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Scala: escalar

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Scala: elasticsearch-client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Smalltalk: elasticsearch-smalltalk

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Swift: Elasticsearch NIO Client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Vert.x: elasticsearch-client

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

Spring Data Elasticsearch

I currently use
I don’t use, but I am interested in using
I don’t use, and I am not interested in using

0 voters

robcowart · June 19, 2021, 5:38am

You missed… “I use, but wish I had a better alternative”

robcowart · June 19, 2021, 5:49am

Somethings that may be useful to add…

Telegraf - Has an Elasticsearch output and is useful as an alternative to Metricbeat, especially for data sources not supported by Metricbeat or other components from Elastic.
Confluent Elasticsearch Connector for Kafka Connect - While Kafka is on the list, there are lots of ways to use Kafka without Kafka Connect. Personally I am not a fan of Connect and the Elasticsearch Connector, but it is somewhat widely used.
StreamSets and NiFi - I think of both of these as “Logstash with a GUI”. However like most GUIs, they are great for beginners, but eventually they just get in the way… at least IMO.

jkowall · June 20, 2021, 2:05am

Good additions, we have customers using Telegraf, but we are using M3DB backend for time series

ralph · June 21, 2021, 6:32am

hm, i voted “yes” on Kafka - but we’re not using Kafka Connect, we have our own service in-between which consumes the Kafka messages from the topics we’re interested in, applies transformations and then writes this into ES. should i’ve voted “no”? if so, maybe rename “Kafka” to “Kafka Connect”?

alexz00 · June 21, 2021, 9:27am

As additional client I would add the Hadoop/Apache Spark Elasticsearch connector: Elasticsearch for Apache Hadoop [8.4] | Elastic

elifish · June 21, 2021, 2:58pm

If you are using Kafka at all in your deployment topology to eventually write to ES or OpenSearch, yes is the way to vote.

elifish · June 21, 2021, 2:59pm

Thanks for sharing these details! I’ll note them down!

hagayg · June 22, 2021, 2:55am

. Net seems to be missing the NEST library, the elasticsearch client is the lower level one (same as Java)

SergioFG · June 24, 2021, 10:55am

Not sure if these two tools fit in with the rest, but we use ElastiFlow for netflow data and Wazuh as a SIEM

elifish · June 29, 2021, 3:03pm

Closing the poll today. Thanks everyone for filling out the poll!

Topic		Replies	Views
Cluster monitoring OpenDistro	36	7299	August 23, 2021
How to send logs kafka to opensearch OpenSearch	32	4387	November 25, 2021
OpenDistro and logstash / filebeats Open Source Elasticsearch and Kibana	1	672	February 7, 2023
Elastic Agent equivalent in Opensearch OpenSearch feature-request	1	1230	December 12, 2023
Best agent to use OpenSearch	6	647	June 28, 2024

What Clients, Agents, and Ingestion Tools do you use?

Agents

Ingestion Aggregators

Data Streaming

Clients

Related topics