OpenSearch on ELK stack

General Feedback
1

Hello,

With my team, we’ve start a log centralization POC in an industrial sector (10000 devices with some old OS).

No cloud, only on premise.

Plan A : use ELK.

After a few research, troubles poping:

ELK is no more opensource

Price can go high with lot of nodes

ELK make you glued with elastic.co

Plan B :

Agent (Logstash, beat, rsylog…) => OpenSearch => DashBoard/Kibana

Does OpenSearch is a good alternative to ES in replacement of a ELK stack ?

What problems will I have to deal with ?

Thanks in advance

2

For log aggregration I’d say ES and OS are equivalent. We are doing a similar POC with Kafka in the middle to gain flexibility / abstraction.

fluent-bit host → fluent-bit ingress → kafka → dataprepper → opensearch

is the pipeline we are using now. For log collectors it’s a mix of filebeat(not my preferred option, licensing)/fluent-bit. For ingress it’s a mix of fluent-bit and logstash-oss (not my preferred option, licensing). Our central opensearch cluster is just one of the possible destinations, that’s why kafka can be useful.

3

Thanks a lot, I really appreciate.

Your POC is really interresting.

I will only use OpenSearch, in my case, Kafka is useless?

Why using licencing agent (beat) instead of OpenSource ones (RSyslog, SyslogNG, Syslogd, Syslog-Win32…)?

I though that Fluent-bit was only for cloud and containerized environments. Does it works with “classic” hardware?

4

I’m a simple man, I read rsyslog and I reply :sweat_smile:

Jokes aside:

  • you can get the OSS version of Filebeat, too: https://www.elastic.co/downloads/past-releases#filebeat-oss
  • whether you want Kafka or not in your pipeline is a separate discussion to whether you can use OpenSearch or not. For the latter, I’d say yes. For Kafka, I’d use it if I’d need a central buffer (e.g. if I don’t have an agent that can buffer a lot, like Filebeat, unlike rsyslog). Kafka also helps if you need replay capabilities (e.g. as a short-term backup)
  • fluent-bit can work on its own (i.e. outside containers)
5

Thanks a lot.

Just install openSearch and try to play with it.

We’re planned to use this solution for loggin.

6

You’re welcome.

If you need something for logs and don’t want to worry about the pipeline, check out our logging SaaS. We expose an OpenSearch API, so things like Filebeat OSS and FluentBit can send natively. Any syslog is also natively supported. And we take care of Kafka, you can also define pipelines (e.g. grok, anonymize) on the server side, we take care of shards, scaling, all that. And we expose APIs for search, too (besides the UI), so if you need custom scripts you can go ahead :slight_smile:

Just thought it might be useful, sorry for the plug. If you need any help with OpenSearch&friends, let me know.