Compute similarty score while ingesting data


I have a pipeline which contains :

  • a file parser that embedds some text fileds in vectors
  • an elasticsearch ingest pipeline that process additionnel transformations on other numerical/date fields

I’d like to use Knn for a classification use case. i.e. computing the similarity score of each embedded vector against a set of labelised vectors and adding a label to this vector.

Is it possible to do this using some script processor in ingest pipeline ?

@wassim.dhib apologies for responding late. I believe script processor is used to perform an operation on individual document like add fields, replace score, etc. from same index.
In your case, I will build another pipeline as , for every document which contains embedded vector, i will use those vector as query vector for search query on index containing labelised vectors , calculate the label from the search result and update document with label.