So we are storing logs in Opensearch and we would like a way to stream the logs to the end user. As of now we just use the search API, search the logs and then show them in the UI but we are working to make it more real time.
If Opensearch does not support streaming, are there any solutions to how we can implement a solution?
I have thought about this also for my application but I haven’t done any work on it. I think real time feed via the OpenSearch API is limited by the refresh interval of OpenSearch, which is configurable.
You could try putting some kind of reverse proxy in front of OpenSearch with some custom code to also serve a WebSocket endpoint, and then stream all the incoming log ingest lines over to any WebSocket connections. It would not get the benefit of any ingest pipelines in OpenSearch however, and you would have to make sure there is proper authentication.
I would be interested in hearing if you are successful with any better solution.
Depending on how you have things set up, it looks like there might be a live log view functionality somewhere in the observability pages. I wasn’t able to get it to work for my data.