versions: 2.16
Hi All, i use vector to index into opensearch. I appear to have the odd message that is making its way through which is not meeting the needs for a data stream index - likely missing timestamp. I am unable to find how that is getting through the flow at the moment.
Can opensearch log the message vector is posting that it does not like ? I am posting in batches so difficult to see in vector.
Hi @coredump17 ,
How do you ingest data? Do you use any ingestion tools?
To see more detailed OpenSearch logs, you can try switching the log level to debug or trace mode:
hi @Eugene7 ,
i use vector.dev (vrl) to ingest many different log formats via a kafka feed.
Could you please send an example of your ingest data? Do you use k-NN vectors in your OpenSearch cluster to store the data? Do you have any error massages in the OpenSearch logs?
According to the documentation here, you must have a timestamp field for a data stream object. But it’s possible to define your own custom timestamp parameter in the data_stream object. There are other ways to ingest ingest data with no required timestamp parameter. Please have a look at the following documentation:
Hi. Thanks for your time. It would appear the issue is with my interpretation of vector.dev VRL. Remap with VRL | Vector documentation is set to false so we had some timestamps not in the correct format but passing the regex test and then leaving it up to opensearch to reject.