Just starting with opensearch and want to send some logs from my webserver to opensearch server.
I have setup fluentbit on the webserver and was under the assumption that I could directly send my logs to opensearch via the opensearch plugin from fluentbit (OpenSearch - Fluent Bit: Official Manual)
But I alo have read about dataprepper being the way to go for opensearch to receive data.
What is the best way to proceed? Is dataprepper necessary? My goal is to keep it as simple as possible without too many components.
Thanks in advance for any insights.
It depends what you like for a log shipper. I notice with DataPrepper I created a port needed ( i.e, source). You can ship those logs directly to Opensearch.
One or more sinks
(Optional) One buffer
(Optional) One or more processors
Fluent-bit I havent worked with that much.
Logstash I know better, you create multiply-inputs per-configuration. The example below all my beats send logs to logstash and it gets processed before hitting Opensearch. I believe Data-prepper can do the same but I havent worked with it that much.
# Beats -> Logstash -> OpenSearch pipeline.
port => 5044
tags => [ 'beat' ]
port => 5144
tags => ['syslog']
port => 12345
tags => ['fluent']
I know All three (i.e., Logstash, Fluent-bit & Data-Prepper) can send logs directly to OpenSearch.
As for one of the beats I havent tried that yet.
Yesterday I manageed to get it working with only fluent-bit and opensearch. In case it helps anybody here is my setup:
My docker-compose.yml file:
- bootstrap.memory_lock=true # along with the memlock settings below, disables swapping
- "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # minimum and maximum Java heap size, recommend setting both to 50% of system RAM
soft: 65536 # maximum number of open files for the OpenSearch user, set to at least 65536 on modern systems
- 9600:9600 # required for Performance Analyzer
# Read interval (sec) Default: 1
#I created an open search index with the command
curl -X PUT "https://localhost:9200/single_index?pretty" -ku 'admin'
systemctl start fluent-bit
I was getting
broken connection to localhost:9200 ? from fluentbit and in the opensearch logs I saw
NotSslRecordException: not an SSL/TLS record so I added
Then I got
Action/metadata line  contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line  contains an unknown parameter [_type]"},"status":400}
which was solved by adding
I guess the Type parameter does not need to be there anymore. Anyways this whole thing was a bit of rough testing anyway…
Then data was coming in!
Awesome and BTW thanks for sharing.
agree!! Thanks @Gsmitt for the assist, and thanks @onehigh for coming back to let the community know about the resolution!