Knn and neural search pipeline

OpenSearch
1

Versions (relevant - OpenSearch/Dashboard/Server OS/Browser): Opensearch 2.12.0

I have setup a pipeline with a connector using openai from one of the available blueprints.

When I run the mapping and then index some docs - it has created a separate field with a type of float as well as my knn field so when I try and search I get an error along the lines of this field is not knn_type. The float field seems to get created when I add the pipeline. When indexing it works fine i.e. the vectors get added to the knn field but the searches always give me and error.

I have followed the documentation as much as possible but cannot see what I am doing wrong

Configuration:

Relevant Logs or Screenshots:

2

this is the mapping

{
"test": {
"aliases": {},
"mappings": {
"properties": {
"all_embedding": {
"type": "float"
},
"all_text": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"mappings": {
"properties": {
"all_embedding": {
"type": "knn_vector",
"dimension": 768
},
"all_text": {
"type": "text"
},
"body_text": {
"type": "text"
},
"description_text": {
"type": "text"
},
"firstcreated": {
"type": "date",
"format": "date_time_no_millis"
},
"headline_text": {
"type": "text"
},
"issued": {
"type": "date",
"format": "date_time_no_millis"
},
"profile": {
"type": "keyword"
},
"type": {
"type": "keyword"
},
"uri": {
"type": "keyword"
},
"versioncreated": {
"type": "date",
"format": "date_time_no_millis"
}
}
}
}
},
"settings": {
"index": {
"replication": {
"type": "DOCUMENT"
},
"number_of_shards": "4",
"provided_name": "test",
"knn.space_type": "cosinesimil",
"default_pipeline": "openai-pipeline",
"knn": "true",
"creation_date": "1709134533225",
"number_of_replicas": "1",
"uuid": "7o1O7UgNRSq50YsZp0NG0A",
"version": {
"created": "136337827"
}
}
}
}
}
3

this is the search I am trying to do

{
  "_source": {
    "excludes": [
      "all_embedding"
    ]
  },
  "query": {
    "neural": {
      "all_embedding": {
        "query_text": "hello",
        "model_id": "bWlZ8I0BYW0M4P103e2J",
        "k": 5
      }
    }
  }
}
4

there error:

   {
                "type": "query_shard_exception",
                "reason": "failed to create query: Field 'all_embedding' is not knn_vector type.",
                "index": "test",
                "index_uuid": "7o1O7UgNRSq50YsZp0NG0A"
            }
5

it seems like the field that is created by the pipeline is a float - although it looks like a vector when you see it - but when you search it needs to be a knn_type so the search fails with the error above. I have tried this with and openai remote model and a huggingface local model

6

I think I have sorted this now - pilot error - my mapping was being added incorrectly