How to import semi-structured text (like log files) into Opensearch?

OpenSearch
, ,
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):

Describe the issue:

Greetings everyone, I am a new to OpenSearch and my company has set up a single node with version 2.11.0.
I swear that I’ve been reading a lot of documentation, but I am getting confused about the log ingestion part.

I am working on a simple task, which involves importing log files (semi-structured text) from a customer to my OpenSearch. These files are local and unfortunately, I am unable to find the way to do it.

I’ve been reading about log collectors (Fluent Bit) or data prepping, but correct me if I’m wrong, but what I am trying to does not require any of that, because I am not trying to feed OpenSearch via HTTP requests to the API endpoint through my code or some already existing solution.
OpenSearch would be beneficial in centralizing these logs and aiding me in visualization.
I am trying to upload a file that is not even mine to OpenSearch, Is that possible? I can understand that I must normalize the data and so on, but Is it possible to import a local log files, instead of ingesting logs from applications?
Thank you so much

Luci

2

Hi @lulucca

Yes, it’s possible to import local log files. To do that, you can use Fluent Bit with tail input plugin and OpenSearch output plugin . Have you tried setting up Fluent Bit?

3

Hi @Eugene7, thank you so much for your reply. I’ve read about FluentBit as a log collector, but I did not read anything about local log files. I didn’t know that we could install plugins to import local files. This is amazing. I am going to install FluentBit on this new server and experiment with this plugin. I am grateful for your time and assistance.

1 Like