We are pretty new in using OpenSearch / DataPrepper.
We have a swarm environement with fluentbit as logdriver on our container.
We send the data to Dataprepper in order to add them to OpenSearch Nodes.
We would like to split the logs on different address based on the container_name key.
How should we do that?
In the index property, we can only add a dynamic date but nothing else.
Can we add a key in our json with the “target” index?
If we don’t do this, we have all our swarm logs into one unique index, which seems not a good practice.
Currently, we do not have the feature to customize index names by JSON key-value. We’d like to encourage you to open an issue as a feature request in our repository, here’s Data Prepper’s contributing guidelines to help you. Thanks.
Thanks for the feedback.
I saw some issue on this already with feature request that should match this.
But what’s the way to do in the mean time?
Is it a regular pratices to have one big index that contains all containers logs? so when I observe them I must filter first what type of container I want to analyse?
How many containers are you working with?
sorry for the late answer, I was off.
as of now I have 25 different type of container (that can have multiple replica) to monitor but it is just one application. we will grow in future, this is why I wanted to know what are the best practices.
There are two routes that you might consider improve the search experience:
(1) To achieve your customization on index, if Fluent Bit as logdriver is used as a sidecar per container, I would suggest one data-prepper per Fluent Bit client so that you could define your container name in the opensearch index in data-prepper. This would probably depend on what swarm environment you are using
(2) From an opensearch engine perspective, you might consider attach index state management policy to your index to rollover the index when the amount of data passes certain threshold to improve the read/write efficiency. But I am not expert on this. You could post related questions in index state management or opensearch engine division.
Thanks for your input on this.
I will manage with one index until the feature request that has been set for this topic is live.