Is there a way to stream logs from OpenSearch? Any alternative solutions for that?

So we are storing logs in Opensearch and we would like a way to stream the logs to the end user. As of now we just use the search API, search the logs and then show them in the UI but we are working to make it more real time.

If Opensearch does not support streaming, are there any solutions to how we can implement a solution?

1 Like

Hi @ujjawalInfer11 ,

What UI do you use to display logs? Is it OpenSearch Dashboards or something else?

It’s our custom React Dashboard that we are using.

How fast and how frequent do you need to get update?

I think that using OpenSearch API is the fastest way to get updates.

I have thought about this also for my application but I haven’t done any work on it. I think real time feed via the OpenSearch API is limited by the refresh interval of OpenSearch, which is configurable.

You could try putting some kind of reverse proxy in front of OpenSearch with some custom code to also serve a WebSocket endpoint, and then stream all the incoming log ingest lines over to any WebSocket connections. It would not get the benefit of any ingest pipelines in OpenSearch however, and you would have to make sure there is proper authentication.

I would be interested in hearing if you are successful with any better solution.

To use websocket as an output, you can install Logstash with Opensearch input plugin and websocket output plugin:

Interesting, I’ll have to look into that.

Depending on how you have things set up, it looks like there might be a live log view functionality somewhere in the observability pages. I wasn’t able to get it to work for my data.