Can someone please provide the scheduling settings necessary for a Transform to run continuously?
The documentation is unclear what needs to be set. Every time after my transform is caught up with the current data, it returns to the “Disabled” state. I need it to be continuously running to incorporate new data as it is added to the index.
How do we specify which field is used to determine whether a document is included in the transform? In ElasticSearch we would specify the field within the sync portion of the transform definition.
Continuous running of transforms is not currently supported in OpenSearch 1.2, but will be released in 1.3. The pull request to add support for continuous transforms is currently in review over here https://github.com/opensearch-project/index-management/pull/206
You can track the progress and contribute to the review there, if you would like.
Currently, this continuous transforms implementation does not have an equivalent to ElasticSearch’s sync, as a timestamp field does not need to be present in our implementation, so the source_field for each of the groups will solely determine whether a document is included.
Thank you for the info. After searching around I found that it is behaving “as intended” and found the PR. I’m following it intently and looking forward to seeing it released.