Versions : OpenSearch version 2.16 & OpenSearch Dashboards 2.16
Describe the issue: Was trying to explore neural search feature in OpenSearch and ran into some issues when field mappings were defined with nested attributes. Here are the steps to reproduce with OpenSearch dashboard
- Define a pipeline with the following field map information
PUT /_ingest/pipeline/mynested-ingest-pipeline
{
“description”: “An NLP ingest pipeline with nested mapping”,
“processors”: [
{
“text_embedding”: {
“model_id”: “JEABnpEBBvwDk6K90GlZ”,
“field_map”: {
“attributes.Cosmetic Specification.About this item”: “about_item_embedding”,
“attributes.Cosmetic Specification.Product Description”: “description_item_embedding”
}
}
}
]
}
- Create a index with mapping defined as follows. Note that the vector fields defined in the mapping are ‘about_item_embedding’ and ‘description_item_embedding’ both defined as knn_vector. Field mappings for the nested fields are also specified in the request
PUT /my-nested-nlp-index
{
“settings”: {
“index.knn”: true,
“default_pipeline”: “mynested-ingest-pipeline”
},
“mappings”: {
“properties”: {
“type”: {
“type”: “text”
},
“containerName”: {
“type”: “text”
},
“identifier”: {
“type”: “text”
},
“primaryKey”: {
“type”: “text”
},
“displayName”: {
“type”: “text”
},
“about_item_embedding”: {
“type”: “knn_vector”,
“dimension”: 768,
“method”: {
“engine”: “lucene”,
“space_type”: “l2”,
“name”: “hnsw”,
“parameters”: {}
}
},
“description_item_embedding”: {
“type”: “knn_vector”,
“dimension”: 768,
“method”: {
“engine”: “lucene”,
“space_type”: “l2”,
“name”: “hnsw”,
“parameters”: {}
}
},
“attributes”: {
“properties”: {
“Cosmetic Specification”: {
“properties”: {
“About this item”: {
“type”: “text”,
“fields”: {
“keyword”: {
“type”: “keyword”,
“ignore_above”: 256
}
}
},
“Product Description”: {
“type”: “text”,
“fields”: {
“keyword”: {
“type”: “keyword”,
“ignore_above”: 256
}
}
}
}
}
}
}
}
}
}
- Now ingest a document with attributes defined in the mapping
PUT /my-nested-nlp-index/_doc/846051
{
“type”: “ITEM”,
“containerName”: “Cosmetic Catalog”,
“identifier”: “846051”,
“primaryKey”: “COSM00975”,
“displayName”: “Studio HD Mattifying Cream Foundation”,
“attributes”: {
“Cosmetic Specification”: {
“Rating”: “4.00”,
“Product Description”: “
This high-definition, oil-free, lightweight foundation is perfect for normal to oily skin types. The cream-to-powder formula minimizes fine lines and leaves a natural, long-lasting, matte finish. Includes a compact sponge for easy application and portable beauty.Shade Range
”,“About this item”: “Matte liquid foundation for normal to oily skin, Blurs pores for a natural look, Contains SPF 22 for sun protection Provides a natural, matte finish, and stops shine, Lightweight and comfortable to wear without caking Simply apply foundation onto skin and blend with fingertips, foundation brush, or makeup sponge”,
“Disclaimer”: [
“Vegan”,
“Natural”,
“Oil free”
],
“Brand”: “Cosmetic Brands-DIOR”,
“Product Type”: “Foundation”
}
}
}
- Check the document to see if the vector fields are generated. Please note that the response generated is missing vector information for vector field ‘about_item_embedding’ (This also seems like a defect as only one nested mapping is created). Vector data for field ‘description_item_embedding’ truncated for brevity.
GET /my-nested-nlp-index/_doc/846051
{
“_index”: “my-nested-nlp-index”,
“_id”: “846051”,
“_version”: 1,
“_seq_no”: 0,
“_primary_term”: 1,
“found”: true,
“_source”: {
“identifier”: “846051”,
“containerName”: “Cosmetic Catalog”,
“displayName”: “Studio HD Mattifying Cream Foundation”,
“attributes”: {
“Cosmetic Specification”: {
“Brand”: “Cosmetic Brands-DIOR”,
“Product Type”: “Foundation”,
“Rating”: “4.00”,
“About this item”: “Matte liquid foundation for normal to oily skin, Blurs pores for a natural look, Contains SPF 22 for sun protection Provides a natural, matte finish, and stops shine, Lightweight and comfortable to wear without caking Simply apply foundation onto skin and blend with fingertips, foundation brush, or makeup sponge”,
“Disclaimer”: [
“Vegan”,
“Natural”,
“Oil free”
],
“Product Description”: “
This high-definition, oil-free, lightweight foundation is perfect for normal to oily skin types. The cream-to-powder formula minimizes fine lines and leaves a natural, long-lasting, matte finish. Includes a compact sponge for easy application and portable beauty.Shade Range
”,“description_item_embedding”: [
0.1705151,
-0.18456638,
0.008977836,
…
]
}
},
“type”: “ITEM”,
“primaryKey”: “COSM00975”
}
}
- Perform a neural search query for the vector field that is generated. There are no documents found (zero hits)
GET /my-nested-nlp-index/_search
{
“query”: {
“neural”: {
“description_item_embedding”: {
“query_text”: “oil free”,
“model_id”: “JEABnpEBBvwDk6K90GlZ”,
“k”: 5
}
}
}
}
Response:
{
“took”: 24,
“timed_out”: false,
“_shards”: {
“total”: 1,
“successful”: 1,
“skipped”: 0,
“failed”: 0
},
“hits”: {
“total”: {
“value”: 0,
“relation”: “eq”
},
“max_score”: null,
“hits”:
}
}
I have tried quite a few combinations but this doesn’t seem to work. Please share any suggestion or information that you believe can address the issue.