Hi,
I am having an OpenSearch on-prem instance running on Docker.
Currently logstash container feeds log files to opensearch.
But there is requirement to feed the logs to opensearch via APIs as the access to log files is not possible.
My question is, is it possible to use API to feed log files to Opensearch?
Please provide the example or document reference if that is available.
Hi @tpatil30
Absolutely. There are several apps specifically meant to follow log files and feed them into OpenSearch for some real time observability into your system.
The one that immediately comes to mind is fluentbit. In a large widely distributed system, fluentbit would usually be used in conjunction with fluentd to offer a single point of collection for your opensearch cluster. If you’re just starting out, a single instance of fluentbit is just what you need.
I would start off here: Fluent Bit v1.9 Documentation - Fluent Bit: Official Manual
The general idea is this - you configure fluentbit on whatever system has the logs you want to ingest. I think if you’re hoping to grab existing log files, you would want to configure fluentbit using the tail plugin - Tail - Fluent Bit: Official Manual
You basically point it at a file and it will watch it and submit it to the opensearch endpoint you have configured.
There’s a lot of input types for fluentbit. Take your time exploring them all. If you have a lot of apps using syslog instead of appending to a file, there’s also a syslog input plugin you can use. Usually a fully configured production install takes inputs from all over the place, but you’ll have to flavor to your own needs.
Please let me know if there’s further references I can provide you, or if you need help coming up with a configuration file.
Happy searching!
Nate
Hi Nate,
Thank you for your response.
I will go through the Fluent Bit documentation and do some exploration.
BR,
Tushar
Hi Nate,
I saw the Fluent Bit documentation.
Fluent Bit and Logstash are similar tools which feeds log files to OpenSearch.
Is there any way to feed log files directly to OpenSearch without such tools installed on the system and using any API?
Thanks
@tpatil30 - I’m afraid not. The very nature of the underlying software is such that the API is the only way to feed documents into OpenSearch.
Could you describe the solution you were hoping for? Maybe we could work together to file a feature request.
Nate
Hi @nateynate
I am also having the similar requirement like tpatil30.
Instead of installing FluentBit or any other agent, I need to send the data to opensearch using API, so would like to know whether this is possible?
If yes, could you please share an example.
On the opensearch dashboard home page, I can see the Opensearch API as shown below, but I am not able to pass a logfile using this API console in opensearch.
I think I see what’s going on. I’ll try to elaborate for you.
The screen you’re looking at there is called the “Dev Console” - it’s meant to be a helpful tool to remind you with auto completion what an API call would look like, and also to perform small manual operations. It’s not technically the API, although when you create a document with it it talks to the API for you. I’m afraid that through the Dev Console there, there’s no such way to just hand it a log file and have it populate an index.
Have a look at the introduction docs to indexing in general: Index data - OpenSearch documentation
These problems are much easier solved by using freely available tools that talk to the API for you that you can configure to break the pieces of your log lines up into usable fields that you can aggregate.
If you’re still hesitant to install an agent to communicate with the API for you, you’ll have to write your own client. Ingesting data is really meant to be done via a sidecar kind of app like fluentbit, fluentd, logstash, data prepper, etc.
We do provide an OpenSearch SDK for a number of programming languages that you can use to ingest your logs with a bit of code, although, again, this is the exact problem those agents above are meant to solve.
OpenSearch only indexes what it’s fed via HTTP requests to the API endpoint (usually https://localhost:9200/) via your own code or some already existing solution. You could even use curl if you like. If you forget what the payload of an API call is supposed to look like, use the Dev Console for payload syntax help.
Sorry if that was a novel and you were just expecting yes or no. 
The short answer is no - you can’t ingest and index a log file from the Dev Console window.
Nate
Hi Nate,
Thanks for your reply.
However we found something related to feeding logs to kibana through APIs in these articles:
We want to send logs from SAP Cloud (CPI) to OpenSearch through API and we are looking for a solution for that.
Thanks.
It looks like these examples actually use some kind of scripting language called “Groovy” to send the data to LogStash in the form of XML.
This might actually be of use to you - LogStash has an OpenSearch plugin. Meaning, that if you could get just the section that sends data to LogStash working, you could use the OpenSearch Logstash plugin to use OpenSearch and OpenSearch Dashboards instead of the source restricted Elastic and Kibana.
I’m afraid I’m unfamiliar with the SAP CPI that the authors use, but I have asked around to see if I can find someone who can chime in here.
Nate
