What is the best approach for such scenario.
I have different log sources: syslog (different flavors), some other syslog like sources) which are sent from fluentbit.
I need to create different indices for each such source.
I use grok to structure them.
The problem is how to implement above steps in dataprepper?
The core problem is how to check for the source log type to sink it into propper opensearch index?
I can setup fluentbit to send each log type to different dataprepper pipeline, but that requires each pipeline to listen on different port. I don’t currently see any other option.
I could use grok behavior which matches the patterns and ends processing after first success. But after this I have no option to set (with some metadata for example) what grok actually matched to use later this info with conditional routing feature wich would choose the right sink.
Any suggestions?
P.