Kafka sink to OpenSearch

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser):
1.3

Describe the issue:
I am trying to follow this documentation to ingest logs via kafka sink to opensearch - Create a sink connector from Apache Kafka® to OpenSearch® - Aiven

I see that - "The current version of the OpenSearch sink connector is not able to automatically create daily indices in OpenSearch. Therefore you need to create the indices with the correct name before starting the sink connector. "

How is it possible to create indices in advance. If I have ALB logs being generated everyday, do I have to create alb-mar21, alb-mar22, alb-mar23, alb-mar24 well in advance (prolly a week or month back) so ingestion can happen?

Can someone clarify?

Configuration:

Relevant Logs or Screenshots:

2

@searchwithme you could use index templates [1] to create daily indices, thank you.

[1] Index templates - OpenSearch documentation

3

But the doc says " The current version of the OpenSearch sink connector is not able to automatically create daily indices in OpenSearch. Therefore you need to create the indices with the correct name before starting the sink connector. " ".

will a new index still be created everyday prior to ingestion from kafka?

4

So AFAIK this is how it works (in case when index does not exist):

  • the connector check if index (or data stream) is present or not
  • if index (or data stream) is not present, it tries to create one but without any mappings
  • here the templates come handy - the mappings will be provided based on index name pattern
1 Like